{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <center> Semantic Relation Classification via Bidirectional LSTM Networks w/ Entity-aware Attention using Latent Entity Typing </center>\n",
    "\n",
    "# <center> Tensorflow Estimator </center>\n",
    "\n",
    "This repository contains the official TensorFlow implementation of the following paper:\n",
    "\n",
    "> **Semantic Relation Classification via Bidirectional LSTM Networks with Entity-aware Attention using Latent Entity Typing**<br>\n",
    "> Joohong Lee, Sangwoo Seo, Yong Suk Choi<br>\n",
    "> [https://arxiv.org/abs/1901.08163](https://arxiv.org/abs/1901.08163)\n",
    "> \n",
    "> **Abstract:** *Classifying semantic relations between entity pairs in sentences is an important task in Natural Language Processing (NLP). Most previous models for relation classification rely on the high-level lexical and syntactic features obtained by NLP tools such as WordNet, dependency parser, part-of-speech (POS) tagger, and named entity recognizers (NER). In addition, state-of-the-art neural models based on attention mechanisms do not fully utilize information of entity that may be the most crucial features for relation classification. To address these issues, we propose a novel end-to-end recurrent neural model which incorporates an entity-aware attention mechanism with a latent entity typing (LET) method. Our model not only utilizes entities and their latent types as features effectively but also is more interpretable by visualizing attention mechanisms applied to our model and results of LET. Experimental results on the SemEval-2010 Task 8, one of the most popular relation classification task, demonstrate that our model outperforms existing state-of-the-art models without any high-level features.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![title](https://user-images.githubusercontent.com/15166794/52579582-c7339100-2e69-11e9-9081-711e7576e717.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Code was transformed into an estimator format from the following repository:\n",
    "\n",
    "> **Entity-aware Attention for Relation Classification**<br>    \n",
    "> [https://github.com/roomylee/entity-aware-relation-classification](https://github.com/roomylee/entity-aware-relation-classification)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Required Libraries \n",
    "\n",
    "Please install them manually using conda or pip."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/luis.magana/anaconda3/lib/python3.7/site-packages/tqdm/autonotebook/__init__.py:14: TqdmExperimentalWarning: Using `tqdm.autonotebook.tqdm` in notebook mode. Use `tqdm.tqdm` instead to force console mode (e.g. in jupyter console)\n",
      "  \" (e.g. in jupyter console)\", TqdmExperimentalWarning)\n",
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "from tensorflow import data\n",
    "import pandas as pd\n",
    "import swifter\n",
    "from datetime import datetime\n",
    "from keras.preprocessing.text import Tokenizer\n",
    "import multiprocessing\n",
    "import shutil\n",
    "import re\n",
    "import nltk\n",
    "from tqdm import tqdm_notebook\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Global Variables  and FLAGS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "MODEL_NAME = 'EARC-model-01'\n",
    "TRAIN_DATA_FILES_PATTERN = 'data/processed/train-*.tsv'\n",
    "VALID_DATA_FILES_PATTERN = 'data/processed/valid-*.tsv'\n",
    "VOCAB_LIST_FILE = 'data/processed/vocab_list.tsv'\n",
    "POS_LIST_FILE = 'data/processed/posidx_list.tsv'\n",
    "RESUME_TRAINING = True\n",
    "MULTI_THREADING = True\n",
    "\n",
    "CLASS_TO_LABEL = {'Other': 0,\n",
    "                  'Message-Topic(e1,e2)': 1, 'Message-Topic(e2,e1)': 2,\n",
    "                  'Product-Producer(e1,e2)': 3, 'Product-Producer(e2,e1)': 4,\n",
    "                  'Instrument-Agency(e1,e2)': 5, 'Instrument-Agency(e2,e1)': 6,\n",
    "                  'Entity-Destination(e1,e2)': 7, 'Entity-Destination(e2,e1)': 8,\n",
    "                  'Cause-Effect(e1,e2)': 9, 'Cause-Effect(e2,e1)': 10,\n",
    "                  'Component-Whole(e1,e2)': 11, 'Component-Whole(e2,e1)': 12,\n",
    "                  'Entity-Origin(e1,e2)': 13, 'Entity-Origin(e2,e1)': 14,\n",
    "                  'Member-Collection(e1,e2)': 15, 'Member-Collection(e2,e1)': 16,\n",
    "                  'Content-Container(e1,e2)': 17, 'Content-Container(e2,e1)': 18}\n",
    "\n",
    "TARGET_LABELS = list(CLASS_TO_LABEL.keys())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "MAX_DOCUMENT_LENGTH = 90\n",
    "PAD_WORD = '#=KS=#'\n",
    "HEADER_DEFAULTS = [['NA'], ['NA'], ['NA'], ['NA'], ['NA'], ['NA']]\n",
    "TARGET_NAME = 'class'\n",
    "WEIGHT_COLUNM_NAME = 'weight' #This will not be used for now"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Text Cleaning & Feature Engineering\n",
    "\n",
    "A text input needs to be transformed, using ```text2features```. We also need to create a tsv file containing the training data and testing data.\n",
    "\n",
    "```python\n",
    "x1 = \"<e1>Avocados</e1> come from farms in the <e2>south of Mexico</e2>\"\n",
    "\n",
    "text2features(x1, MAX_DOCUMENT_LENGTH)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def clean_str(text):\n",
    "    text = text.lower()\n",
    "    # Clean the text\n",
    "    text = re.sub(r\"[^A-Za-z0-9^,!.\\/'+-=]\", \" \", text)\n",
    "    text = re.sub(r\"what's\", \"what is \", text)\n",
    "    text = re.sub(r\"that's\", \"that is \", text)\n",
    "    text = re.sub(r\"there's\", \"there is \", text)\n",
    "    text = re.sub(r\"it's\", \"it is \", text)\n",
    "    text = re.sub(r\"\\'s\", \" \", text)\n",
    "    text = re.sub(r\"\\'ve\", \" have \", text)\n",
    "    text = re.sub(r\"can't\", \"can not \", text)\n",
    "    text = re.sub(r\"n't\", \" not \", text)\n",
    "    text = re.sub(r\"i'm\", \"i am \", text)\n",
    "    text = re.sub(r\"\\'re\", \" are \", text)\n",
    "    text = re.sub(r\"\\'d\", \" would \", text)\n",
    "    text = re.sub(r\"\\'ll\", \" will \", text)\n",
    "    text = re.sub(r\",\", \" \", text)\n",
    "    text = re.sub(r\"\\.\", \" \", text)\n",
    "    text = re.sub(r\"!\", \" ! \", text)\n",
    "    text = re.sub(r\"\\/\", \" \", text)\n",
    "    text = re.sub(r\"\\^\", \" ^ \", text)\n",
    "    text = re.sub(r\"\\+\", \" + \", text)\n",
    "    text = re.sub(r\"\\-\", \" - \", text)\n",
    "    text = re.sub(r\"\\=\", \" = \", text)\n",
    "    text = re.sub(r\"'\", \" \", text)\n",
    "    text = re.sub(r\"(\\d+)(k)\", r\"\\g<1>000\", text)\n",
    "    text = re.sub(r\":\", \" : \", text)\n",
    "    text = re.sub(r\" e g \", \" eg \", text)\n",
    "    text = re.sub(r\" b g \", \" bg \", text)\n",
    "    text = re.sub(r\" u s \", \" american \", text)\n",
    "    text = re.sub(r\"\\0s\", \"0\", text)\n",
    "    text = re.sub(r\" 9 11 \", \"911\", text)\n",
    "    text = re.sub(r\"e - mail\", \"email\", text)\n",
    "    text = re.sub(r\"j k\", \"jk\", text)\n",
    "    text = re.sub(r\"\\s{2,}\", \" \", text)\n",
    "    return text.strip()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def text2features(text, max_sentence_length):\n",
    "    text = text.replace('<e1>', ' _e11_ ')\n",
    "    text = text.replace('</e1>', ' _e12_ ')\n",
    "    text = text.replace('<e2>', ' _e21_ ')\n",
    "    text = text.replace('</e2>', ' _e22_ ')\n",
    "    text = clean_str(text)\n",
    "    tokens = nltk.word_tokenize(text)\n",
    "    e1 = tokens.index(\"e12\") - 1\n",
    "    e2 = tokens.index(\"e22\") - 1\n",
    "    p1 = \"\"\n",
    "    p2 = \"\"\n",
    "    for word_idx in range(len(tokens)):\n",
    "        p1 += str((max_sentence_length - 1) + word_idx - e1) + \" \"\n",
    "        p2 += str((max_sentence_length - 1) + word_idx - e2) + \" \"\n",
    "    text = \" \".join(tokens)\n",
    "    features = {\n",
    "        'text': text, \n",
    "        'e1': e1,\n",
    "        'e2': e2,\n",
    "        'p1': p1,\n",
    "        'p2': p2\n",
    "    }\n",
    "    return features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "996901dd660940db863bb7f837035614",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Pandas Apply', max=8000, style=ProgressStyle(description_widt…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "df = pd.read_csv('./data/raw-train-data.tsv', sep='\\t', header=None, names=['class', 'text'])\n",
    "s = df.text.swifter.apply(lambda x: text2features(x, MAX_DOCUMENT_LENGTH))\n",
    "train = pd.DataFrame(s.to_list())\n",
    "train['class'] = df['class']\n",
    "HEADER = train.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4e79dd4af1564ce4bac417b1266b99e1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Pandas Apply', max=2717, style=ProgressStyle(description_widt…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "df = pd.read_csv('./data/raw-valid-data.tsv', sep='\\t', header=None, names=['class', 'text'])\n",
    "s = df.text.swifter.apply(lambda x: text2features(x, MAX_DOCUMENT_LENGTH))\n",
    "test = pd.DataFrame(s.to_list())\n",
    "test['class'] = df['class']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "train.to_csv('./data/processed/train-data.tsv', sep='\\t', index=False, header=False)\n",
    "test.to_csv('./data/processed/valid-data.tsv', sep='\\t', index=False, header=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tokenizing input data...\n",
      "dictionary size:  22382\n"
     ]
    }
   ],
   "source": [
    "print(\"tokenizing input data...\")\n",
    "tokenizer = Tokenizer(lower=False, char_level=False)\n",
    "tokenizer.fit_on_texts(train.text.tolist() + test.text.tolist())  #leaky\n",
    "word_index = tokenizer.word_index\n",
    "print(\"dictionary size: \", len(word_index))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "vocab = list(word_index.keys())\n",
    "vocab.insert(0, '#=KS=#')\n",
    "N_WORDS = len(vocab)\n",
    "with open('./data/processed/vocab_list.tsv', 'w+') as f:\n",
    "    for w in vocab:\n",
    "        f.write(w + '\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "pos_tokenizer = Tokenizer(lower=False, char_level=False)\n",
    "pos_tokenizer.fit_on_texts(train.p1.tolist() + train.p2.tolist() + test.p1.tolist() + test.p2.tolist())  #leaky"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "pos_idx = list(pos_tokenizer.word_index.keys())\n",
    "pos_idx.insert(0, '0')\n",
    "N_POS = len(pos_idx)\n",
    "with open('./data/processed/posidx_list.tsv', 'w+') as f:\n",
    "    for p in pos_idx:\n",
    "        f.write(p + '\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build Glove Embedding Matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_coefs(word, *arr):\n",
    "    return word, np.asarray(arr, dtype='float32')\n",
    "\n",
    "def load_embeddings(path):\n",
    "    with open(path) as f:\n",
    "        return dict(get_coefs(*line.strip().split(' ')) for line in tqdm_notebook(f))\n",
    "\n",
    "def build_matrix(word_index, path):\n",
    "    embedding_index = load_embeddings(path)\n",
    "    embedding_matrix = np.zeros((len(word_index) + 1, 300))\n",
    "    unknown_words = []\n",
    "    \n",
    "    for word, i in word_index.items():\n",
    "        try:\n",
    "            embedding_matrix[i] = embedding_index[word]\n",
    "        except KeyError:\n",
    "            unknown_words.append(word)\n",
    "    return embedding_matrix, unknown_words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "FAST_TEXT = '/home/luis.magana/embeddings/crawl-300d-2M.vec'\n",
    "GLOVE = '/home/luis.magana/embeddings/glove.840B.300d.txt'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e44f04ef27804b5483254468e4e21c22",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=1, bar_style='info', max=1), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "embedding_matrix, _ = build_matrix(word_index, GLOVE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Estimator Functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "def parse_tsv_row(tsv_row):\n",
    "    \"\"\"\n",
    "    This function assumes that the text has already been cleaned and tokenized. \n",
    "    \"\"\"\n",
    "    data = tf.decode_csv(tsv_row, record_defaults=HEADER_DEFAULTS, field_delim='\\t')\n",
    "    features = dict(zip(HEADER, data))\n",
    "    target = features.pop(TARGET_NAME)\n",
    "    # giving more weight to \"spam\" records are the are only 13% of the training set\n",
    "    # features[WEIGHT_COLUNM_NAME] =  tf.cond( tf.equal(target,'spam'), lambda: 6.6, lambda: 1.0 ) \n",
    "    features[WEIGHT_COLUNM_NAME] = 1 # set to one for now\n",
    "    return features, target"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "def parse_label_2_one_hot(label_string_tensor):\n",
    "    \"\"\"\n",
    "    Takes a tensor string containg the labeled class and returns a one-hot vector representation.\n",
    "    \"\"\"\n",
    "    table = tf.contrib.lookup.index_table_from_tensor(tf.constant(TARGET_LABELS))\n",
    "    index = table.lookup(label_string_tensor)\n",
    "    #one_m_depth =  tf.dtypes.cast(tf.keras.backend.max(index), tf.int32) \n",
    "    #depth = tf.math.add(tf.constant(1), one_m_depth)\n",
    "    return tf.one_hot(index, len(TARGET_LABELS))\n",
    "\n",
    "def input_fn(files_name_pattern, mode=tf.estimator.ModeKeys.EVAL, \n",
    "                 skip_header_lines=0, \n",
    "                 num_epochs=1,\n",
    "                 batch_size=200):\n",
    "    \"\"\"\n",
    "    Input Function for tensorflow estimator. Returns the tensor features and \n",
    "    one-hot representation of the target class.\n",
    "    \"\"\"\n",
    "    shuffle = True if mode == tf.estimator.ModeKeys.TRAIN else False\n",
    "    \n",
    "    num_threads = multiprocessing.cpu_count() if MULTI_THREADING else 1\n",
    "    \n",
    "    buffer_size = 2 * batch_size + 1\n",
    "   \n",
    "    print(\"\\n\", \"* data input_fn:\")\n",
    "    print(\"===========================================\")\n",
    "    print(\"Input file(s): {}\".format(files_name_pattern))\n",
    "    print(\"Batch size: {}\".format(batch_size))\n",
    "    print(\"Epoch Count: {}\".format(num_epochs))\n",
    "    print(\"Mode: {}\".format(mode))\n",
    "    print(\"Thread Count: {}\".format(num_threads))\n",
    "    print(\"Shuffle: {}\".format(shuffle))\n",
    "    print(\"===========================================\", \"\\n\")\n",
    "\n",
    "    file_names = tf.matching_files(files_name_pattern)\n",
    "    dataset = data.TextLineDataset(filenames=file_names)\n",
    "    \n",
    "    dataset = dataset.skip(skip_header_lines)\n",
    "    \n",
    "    if shuffle:\n",
    "        dataset = dataset.shuffle(buffer_size)\n",
    "        \n",
    "    dataset = dataset.map(lambda tsv_row: parse_tsv_row(tsv_row), \n",
    "                          num_parallel_calls=num_threads)\n",
    "    \n",
    "    dataset = dataset.batch(batch_size)\n",
    "    dataset = dataset.repeat(num_epochs)\n",
    "    dataset = dataset.prefetch(buffer_size)\n",
    "    \n",
    "    iterator = dataset.make_one_shot_iterator()\n",
    "    \n",
    "    features, target = iterator.get_next()\n",
    "    return features, parse_label_2_one_hot(target)\n",
    "\n",
    "def process_text(text_feature): \n",
    "    \"\"\"\n",
    "    The text features will be transformed into a word id vector. \n",
    "    \n",
    "    in  ---> ['a misty ridge uprises from the surge <UNK> <UNK> ... <UNK>']\n",
    "    out ---> [27 39 40 41 42  1 43  0  0 ... 0]\n",
    "    \"\"\"\n",
    "    # Load vocabolary lookup table to map word => word_id\n",
    "    vocab_table = tf.contrib.lookup.index_table_from_file(vocabulary_file=VOCAB_LIST_FILE, \n",
    "                                                          num_oov_buckets=1, default_value=-1)\n",
    "    # Split text to words -> this will produce sparse tensor with variable-lengthes (word count) entries\n",
    "    words = tf.string_split(text_feature)\n",
    "    # Convert sparse tensor to dense tensor by padding each entry to match the longest in the batch\n",
    "    dense_words = tf.sparse_tensor_to_dense(words, default_value=PAD_WORD)\n",
    "    # Convert word to word_ids via the vocab lookup table\n",
    "    word_ids = vocab_table.lookup(dense_words)\n",
    "    # Create a word_ids padding\n",
    "    padding = tf.constant([[0,0],[0,MAX_DOCUMENT_LENGTH]])\n",
    "    # Pad all the word_ids entries to the maximum document length\n",
    "    word_ids_padded = tf.pad(word_ids, padding)\n",
    "    word_id_vector = tf.slice(word_ids_padded, [0,0], [-1, MAX_DOCUMENT_LENGTH])\n",
    "    # Return the final word_id_vector\n",
    "    return word_id_vector\n",
    "\n",
    "def process_pos(text_feature):\n",
    "    \"\"\"\n",
    "    The text features will be transformed into a position id vector. See model diagram and paper for more information.\n",
    "    \n",
    "    in  ---> [95 96 97 98 99 100 101 999 999 999 ... 999]\n",
    "    out ---> [11 12 13 14 15  16  21  17  17  17 ...  17]\n",
    "    \"\"\"\n",
    "    # Load vocabolary lookup table to map word => word_id\n",
    "    pos_table = tf.contrib.lookup.index_table_from_file(vocabulary_file=POS_LIST_FILE, \n",
    "                                                          num_oov_buckets=1, default_value=-1)\n",
    "    # Split text to words -> this will produce sparse tensor with variable-lengthes (word count) entries\n",
    "    poss = tf.string_split(text_feature)\n",
    "    # Convert sparse tensor to dense tensor by padding each entry to match the longest in the batch\n",
    "    dense_poss = tf.sparse_tensor_to_dense(poss, default_value='0')\n",
    "    # Convert word to word_ids via the vocab lookup table\n",
    "    pos_ids = pos_table.lookup(dense_poss)\n",
    "    # Create a word_ids padding\n",
    "    padding = tf.constant([[0,0],[0, MAX_DOCUMENT_LENGTH]])\n",
    "    # Pad all the word_ids entries to the maximum document length\n",
    "    pos_ids_padded = tf.pad(pos_ids, padding)\n",
    "    pos_id_vector = tf.slice(pos_ids_padded, [0,0], [-1, MAX_DOCUMENT_LENGTH])\n",
    "    # Return the final word_id_vector\n",
    "    return pos_id_vector\n",
    "\n",
    "def process_ent(text_feature):\n",
    "    \"\"\"\n",
    "    The text features will be transformed into tensor of type int32.\n",
    "    \"\"\"\n",
    "    return tf.strings.to_number(text_feature, out_type=tf.dtypes.int32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Custom Attention Blocks for RNNs and Entities\n",
    "\n",
    "These functions were obtained directly from the paper's original code. Please see original author for questions. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "def attention(inputs, e1, e2, p1, p2, attention_size):\n",
    "    # inputs = (batch, seq_len, hidden)\n",
    "    # e1, e2 = (batch, seq_len)\n",
    "    # p1, p2 = (batch, seq_len, dist_emb_size)\n",
    "    # attention_size = scalar(int)\n",
    "    def extract_entity(x, e):\n",
    "        e_idx = tf.concat([tf.expand_dims(tf.range(tf.shape(e)[0]), axis=-1), tf.expand_dims(e, axis=-1)], axis=-1)\n",
    "        return tf.gather_nd(x, e_idx)  # (batch, hidden)\n",
    "    seq_len = tf.shape(inputs)[1]  # fixed at run-time\n",
    "    hidden_size = inputs.shape[2].value  # fixed at compile-time\n",
    "    latent_size = hidden_size\n",
    "\n",
    "    # Latent Relation Variable based on Entities\n",
    "    e1_h = extract_entity(inputs, e1)  # (batch, hidden)\n",
    "    e2_h = extract_entity(inputs, e2)  # (batch, hidden)\n",
    "    e1_type, e2_type, e1_alphas, e2_alphas = latent_type_attention(e1_h, e2_h,\n",
    "                                                                   num_type=3,\n",
    "                                                                   latent_size=latent_size)  # (batch, hidden)\n",
    "    e1_h = tf.concat([e1_h, e1_type], axis=-1)  # (batch, hidden+latent)\n",
    "    e2_h = tf.concat([e2_h, e2_type], axis=-1)  # (batch, hidden+latent)\n",
    "\n",
    "    # v*tanh(W*[h;p1;p2]+W*[e1;e2]) 85.18%? 84.83% 84.55%\n",
    "    e_h = tf.layers.dense(tf.concat([e1_h, e2_h], -1), attention_size, use_bias=False, kernel_initializer=initializer())\n",
    "    e_h = tf.reshape(tf.tile(e_h, [1, seq_len]), [-1, seq_len, attention_size])\n",
    "    v = tf.layers.dense(tf.concat([inputs, p1, p2], axis=-1), attention_size, use_bias=False, kernel_initializer=initializer())\n",
    "    v = tf.tanh(tf.add(v, e_h))\n",
    "\n",
    "    u_omega = tf.get_variable(\"u_omega\", [attention_size], initializer=initializer())\n",
    "    vu = tf.tensordot(v, u_omega, axes=1, name='vu')  # (batch, seq_len)\n",
    "    alphas = tf.nn.softmax(vu, name='alphas')  # (batch, seq_len)\n",
    "\n",
    "    # v*tanh(W*[h;p1;p2;e1;e2]) 85.18% 84.41%\n",
    "    # e1_h = tf.reshape(tf.tile(e1_h, [1, seq_len]), [-1, seq_len, hidden_size+latent_size])\n",
    "    # e2_h = tf.reshape(tf.tile(e2_h, [1, seq_len]), [-1, seq_len, hidden_size+latent_size])\n",
    "    # v = tf.concat([inputs, p1, p2, e1_h, e2_h], axis=-1)\n",
    "    # v = tf.layers.dense(v, attention_size, activation=tf.tanh, kernel_initializer=initializer())\n",
    "    #\n",
    "    # u_omega = tf.get_variable(\"u_omega\", [attention_size], initializer=initializer())\n",
    "    # vu = tf.tensordot(v, u_omega, axes=1, name='vu')  # (batch, seq_len)\n",
    "    # alphas = tf.nn.softmax(vu, name='alphas')  # (batch, seq_len)\n",
    "\n",
    "    # output\n",
    "    output = tf.reduce_sum(inputs * tf.expand_dims(alphas, -1), 1)  # (batch, hidden)\n",
    "\n",
    "    return output, alphas, e1_alphas, e2_alphas\n",
    "\n",
    "\n",
    "def latent_type_attention(e1, e2, num_type, latent_size):\n",
    "    # Latent Entity Type Vectors\n",
    "    latent_type = tf.get_variable(\"latent_type\", shape=[num_type, latent_size], initializer=initializer())\n",
    "\n",
    "    # e1_h = tf.layers.dense(e1, latent_size, kernel_initializer=initializer())\n",
    "    # e2_h = tf.layers.dense(e2, latent_size, kernel_initializer=initializer())\n",
    "\n",
    "    e1_sim = tf.matmul(e1, tf.transpose(latent_type))  # (batch, num_type)\n",
    "    e1_alphas = tf.nn.softmax(e1_sim, name='e1_alphas')  # (batch, num_type)\n",
    "    e1_type = tf.matmul(e1_alphas, latent_type, name='e1_type')  # (batch, hidden)\n",
    "\n",
    "    e2_sim = tf.matmul(e2, tf.transpose(latent_type))  # (batch, num_type)\n",
    "    e2_alphas = tf.nn.softmax(e2_sim, name='e2_alphas')  # (batch, num_type)\n",
    "    e2_type = tf.matmul(e2_alphas, latent_type, name='e2_type')  # (batch, hidden)\n",
    "\n",
    "    return e1_type, e2_type, e1_alphas, e2_alphas\n",
    "\n",
    "\n",
    "def multihead_attention(queries, keys, num_units, num_heads,\n",
    "                        dropout_rate=0, scope=\"multihead_attention\", reuse=None):\n",
    "    with tf.variable_scope(scope, reuse=reuse):\n",
    "        # Linear projections\n",
    "        Q = tf.layers.dense(queries, num_units, kernel_initializer=initializer())  # (N, T_q, C)\n",
    "        K = tf.layers.dense(keys, num_units, kernel_initializer=initializer())  # (N, T_k, C)\n",
    "        V = tf.layers.dense(keys, num_units, kernel_initializer=initializer())  # (N, T_k, C)\n",
    "\n",
    "        # Split and concat\n",
    "        Q_ = tf.concat(tf.split(Q, num_heads, axis=2), axis=0)  # (h*N, T_q, C/h)\n",
    "        K_ = tf.concat(tf.split(K, num_heads, axis=2), axis=0)  # (h*N, T_k, C/h)\n",
    "        V_ = tf.concat(tf.split(V, num_heads, axis=2), axis=0)  # (h*N, T_k, C/h)\n",
    "\n",
    "        # Multiplication\n",
    "        outputs = tf.matmul(Q_, tf.transpose(K_, [0, 2, 1]))  # (h*N, T_q, T_k)\n",
    "\n",
    "        # Scale\n",
    "        outputs /= K_.get_shape().as_list()[-1] ** 0.5\n",
    "\n",
    "        # Key Masking\n",
    "        key_masks = tf.sign(tf.abs(tf.reduce_sum(keys, axis=-1)))  # (N, T_k)\n",
    "        key_masks = tf.tile(key_masks, [num_heads, 1])  # (h*N, T_k)\n",
    "        key_masks = tf.tile(tf.expand_dims(key_masks, 1), [1, tf.shape(queries)[1], 1])  # (h*N, T_q, T_k)\n",
    "\n",
    "        paddings = tf.ones_like(outputs) * (-2 ** 32 + 1)\n",
    "        outputs = tf.where(tf.equal(key_masks, 0), paddings, outputs)  # (h*N, T_q, T_k)\n",
    "\n",
    "        # Activation\n",
    "        alphas = tf.nn.softmax(outputs)  # (h*N, T_q, T_k)\n",
    "\n",
    "        # Query Masking\n",
    "        query_masks = tf.sign(tf.abs(tf.reduce_sum(queries, axis=-1)))  # (N, T_q)\n",
    "        query_masks = tf.tile(query_masks, [num_heads, 1])  # (h*N, T_q)\n",
    "        query_masks = tf.tile(tf.expand_dims(query_masks, -1), [1, 1, tf.shape(keys)[1]])  # (h*N, T_q, T_k)\n",
    "        alphas *= query_masks  # broadcasting. (N, T_q, C)\n",
    "\n",
    "        # Dropouts\n",
    "        alphas = tf.layers.dropout(alphas, rate=dropout_rate, training=tf.convert_to_tensor(True))\n",
    "\n",
    "        # Weighted sum\n",
    "        outputs = tf.matmul(alphas, V_)  # ( h*N, T_q, C/h)\n",
    "\n",
    "        # Restore shape\n",
    "        outputs = tf.concat(tf.split(outputs, num_heads, axis=0), axis=2)  # (N, T_q, C)\n",
    "\n",
    "        # Linear\n",
    "        outputs = tf.layers.dense(outputs, num_units, activation=tf.nn.relu, kernel_initializer=initializer())\n",
    "\n",
    "        # Residual connection\n",
    "        outputs += queries\n",
    "\n",
    "        # Normalize\n",
    "        outputs = layer_norm(outputs)  # (N, T_q, C)\n",
    "\n",
    "    return outputs, alphas\n",
    "\n",
    "\n",
    "def layer_norm(inputs, epsilon=1e-8, scope=\"layer_norm\", reuse=None):\n",
    "    with tf.variable_scope(scope, reuse=reuse):\n",
    "        inputs_shape = inputs.get_shape()\n",
    "        params_shape = inputs_shape[-1:]\n",
    "\n",
    "        mean, variance = tf.nn.moments(inputs, [-1], keep_dims=True)\n",
    "        beta = tf.Variable(tf.zeros(params_shape))\n",
    "        gamma = tf.Variable(tf.ones(params_shape))\n",
    "        normalized = (inputs - mean) / ((variance + epsilon) ** (.5))\n",
    "        outputs = gamma * normalized + beta\n",
    "\n",
    "    return outputs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tensorflow Estimator "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "def initializer():\n",
    "    return tf.keras.initializers.glorot_normal()\n",
    "\n",
    "def seq_length(seq):\n",
    "    relevant = tf.sign(tf.abs(seq))\n",
    "    length = tf.reduce_sum(relevant, reduction_indices=1)\n",
    "    length = tf.cast(length, tf.int32)\n",
    "    return length\n",
    "\n",
    "def model_fn(features, labels, mode, params):\n",
    "    embedding_size = params.embedding_size\n",
    "    pos_vocab_size = params.pos_vocab_size\n",
    "    pos_embedding_size = params.pos_embedding_size\n",
    "    num_heads = params.num_heads\n",
    "    attention_size = params.attention_size\n",
    "    hidden_size = params.hidden_size\n",
    "    l2_reg_lambda = params.l2_reg_lambda\n",
    "    emb_dropout_keep_prob = params.emb_dropout_keep_prob\n",
    "    rnn_dropout_keep_prob = params.rnn_dropout_keep_prob\n",
    "    dropout_keep_prob = params.dropout_keep_prob\n",
    "    num_classes = params.num_classes\n",
    "    learning_rate = params.learning_rate\n",
    "    decay_rate = params.decay_rate\n",
    "    \n",
    "    input_x = process_text(features['text'])\n",
    "    input_e1 = process_ent(features['e1'])\n",
    "    input_e2 = process_ent(features['e2'])\n",
    "    input_p1 = process_pos(features['p1'])\n",
    "    input_p2 = process_pos(features['p2'])\n",
    "    \n",
    "    # Word Embedding Layer\n",
    "    with tf.name_scope('word-embeddings'):\n",
    "        W_text = tf.get_variable(\"W_text\", [N_WORDS, embedding_size], \n",
    "                                    initializer=tf.constant_initializer(params.embedding_initializer()))\n",
    "        embedded_chars = tf.nn.embedding_lookup(W_text, input_x)\n",
    "        #embedded_chars = tf.contrib.layers.embed_sequence(\n",
    "        #    input_x, N_WORDS, embedding_size,\n",
    "        #    initializer=params.embedding_initializer)\n",
    "        \n",
    "    # Position Embedding Layer\n",
    "    with tf.name_scope('position-embeddings'):\n",
    "        W_pos = tf.get_variable(\"W_pos\", [pos_vocab_size, pos_embedding_size], initializer=initializer())\n",
    "        p1 = tf.nn.embedding_lookup(W_pos, input_p1)[:, :tf.shape(embedded_chars)[1]]\n",
    "        p2 = tf.nn.embedding_lookup(W_pos, input_p2)[:, :tf.shape(embedded_chars)[1]]\n",
    "    \n",
    "    # Dropout for Word Embedding\n",
    "    with tf.variable_scope('dropout-embeddings'):\n",
    "        if mode == tf.estimator.ModeKeys.TRAIN:\n",
    "            embedded_chars = tf.nn.dropout(embedded_chars,  emb_dropout_keep_prob)\n",
    "        else:\n",
    "            embedded_chars = tf.nn.dropout(embedded_chars,  1.0)\n",
    "    \n",
    "    # Self Attention\n",
    "    with tf.variable_scope(\"self-attention\"):\n",
    "        attn, alphas = multihead_attention(embedded_chars, embedded_chars,\n",
    "                                                                num_units=embedding_size, num_heads=num_heads)\n",
    "    # Bidirectional LSTM\n",
    "    with tf.variable_scope(\"bi-lstm\"):\n",
    "        #tf.contrib.cudnn_rnn.CudnnCompatibleLSTMCell <--- speed up\n",
    "        #tf.nn.rnn_cell.LSTMCell <--- normal LSTMCell # Must Use Initializer\n",
    "        if mode == tf.estimator.ModeKeys.TRAIN:\n",
    "            rnn_dropout_keep_prob = 1.0\n",
    "        else:\n",
    "            rnn_dropout_keep_prob = params.rnn_dropout_keep_prob\n",
    "        _fw_cell = tf.nn.rnn_cell.LSTMCell(hidden_size, initializer=initializer())\n",
    "        fw_cell = tf.nn.rnn_cell.DropoutWrapper(_fw_cell, rnn_dropout_keep_prob)\n",
    "        _bw_cell = tf.nn.rnn_cell.LSTMCell(hidden_size, initializer=initializer())\n",
    "        bw_cell = tf.nn.rnn_cell.DropoutWrapper(_bw_cell, rnn_dropout_keep_prob)\n",
    "        rnn_outputs, _ = tf.nn.bidirectional_dynamic_rnn(cell_fw=fw_cell,\n",
    "                                                              cell_bw=bw_cell,\n",
    "                                                              inputs=attn,\n",
    "                                                              sequence_length=seq_length(input_x),\n",
    "                                                              dtype=tf.float32)\n",
    "        rnn_outputs = tf.concat(rnn_outputs, axis=-1)\n",
    "        \n",
    "    # Attention\n",
    "    with tf.variable_scope('attention'):\n",
    "        attn, alphas, e1_alphas, e2_alphas = attention(rnn_outputs, input_e1, input_e2,\n",
    "                                                       p1, p2, attention_size=attention_size)\n",
    "    # Dropout\n",
    "    with tf.variable_scope('dropout'):\n",
    "        if mode == tf.estimator.ModeKeys.TRAIN:\n",
    "            h_drop = tf.nn.dropout(attn, dropout_keep_prob)\n",
    "        else: \n",
    "            h_drop = tf.nn.dropout(attn, 1.0)\n",
    "\n",
    "    # Fully connected layer\n",
    "    with tf.variable_scope('output'):\n",
    "        logits = tf.layers.dense(h_drop, num_classes, kernel_initializer=initializer())\n",
    "        probabilities = tf.nn.softmax(logits)\n",
    "        predicted_indices = tf.argmax(logits, 1, name=\"predictions\")\n",
    "        \n",
    "    # Provide an estimator spec for `ModeKeys.PREDICT`.\n",
    "    if mode == tf.estimator.ModeKeys.PREDICT:\n",
    "        # Convert predicted_indices back into strings\n",
    "        predictions = {\n",
    "            'class': tf.gather(TARGET_LABELS, predicted_indices),\n",
    "            'probabilities': probabilities\n",
    "        }\n",
    "        export_outputs = {\n",
    "            'prediction': tf.estimator.export.PredictOutput(predictions)\n",
    "        }\n",
    "        # Provide an estimator spec for `ModeKeys.PREDICT` modes.\n",
    "        return tf.estimator.EstimatorSpec(mode,\n",
    "                                          predictions=predictions,\n",
    "                                          export_outputs=export_outputs)\n",
    "    \n",
    "    # weights\n",
    "    # weights = features[WEIGHT_COLUNM_NAME]\n",
    "    \n",
    "    # Calculate mean cross-entropy loss\n",
    "    losses = tf.nn.softmax_cross_entropy_with_logits_v2(logits=logits, labels=labels)\n",
    "    l2 = tf.add_n([tf.nn.l2_loss(v) for v in tf.trainable_variables()])\n",
    "    loss = tf.reduce_mean(losses) + l2_reg_lambda * l2\n",
    "    loss = tf.identity(loss, name=\"loss\")\n",
    "    tf.summary.scalar(\"loss\", loss)\n",
    "    \n",
    "    if mode == tf.estimator.ModeKeys.TRAIN:\n",
    "        #Calc Train Accuracy\n",
    "        accuracy = tf.metrics.accuracy(labels=tf.argmax(labels, 1), predictions=predicted_indices)\n",
    "        # Save accuracy scalar to Tensorboard output.\n",
    "        tf.summary.scalar('train_accuracy', accuracy[1])\n",
    "        \n",
    "        #Create Hook\n",
    "        summary_hook = tf.train.SummarySaverHook(\n",
    "            save_steps=20,\n",
    "            output_dir='/tmp/tf/train',\n",
    "            summary_op=tf.summary.merge_all())\n",
    "        \n",
    "        # Create Optimiser\n",
    "        optimizer = tf.train.AdadeltaOptimizer(learning_rate, decay_rate, 1e-6)\n",
    "\n",
    "        # Create training operation\n",
    "        gvs = optimizer.compute_gradients(loss)\n",
    "        capped_gvs = [(tf.clip_by_value(grad, -1.0, 1.0), var) for grad, var in gvs]\n",
    "        train_op = optimizer.apply_gradients(capped_gvs, global_step=tf.train.get_global_step())\n",
    "\n",
    "        # Provide an estimator spec for `ModeKeys.TRAIN` modes.\n",
    "        return tf.estimator.EstimatorSpec(mode=mode,\n",
    "                                          loss=loss, \n",
    "                                          train_op=train_op, \n",
    "                                          training_hooks=[summary_hook])\n",
    "          \n",
    "    if mode == tf.estimator.ModeKeys.EVAL: \n",
    "        #Calc Train Accuracy\n",
    "        eval_accuracy = tf.metrics.accuracy(labels=tf.argmax(labels, 1), predictions=predicted_indices)\n",
    "        # Save accuracy scalar to Tensorboard output.\n",
    "        tf.summary.scalar('eval_accuracy', eval_accuracy[1])\n",
    "        \n",
    "        eval_metric_ops = {\n",
    "            'accuracy': eval_accuracy,\n",
    "            'f1_score': tf.contrib.metrics.f1_score(labels, probabilities)\n",
    "        }\n",
    "        # Provide an estimator spec for `ModeKeys.EVAL` modes.\n",
    "        return tf.estimator.EstimatorSpec(mode, \n",
    "                                          loss=loss,\n",
    "                                          eval_metric_ops=eval_metric_ops)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "def embedding_initializer(shape=None, dtype=tf.float32, partition_info=None):\n",
    "    assert dtype is tf.float32\n",
    "    return embedding_matrix\n",
    "\n",
    "def create_estimator(run_config, hparams):\n",
    "    estimator = tf.estimator.Estimator(model_fn=model_fn, \n",
    "                                  params=hparams, \n",
    "                                  config=run_config)\n",
    "    print(\"Estimator Type: {}\".format(type(estimator)))\n",
    "    return estimator"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hyper-Parameter Tunning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('attention_size', 50), ('batch_size', 20), ('decay_rate', 0.9), ('dropout_keep_prob', 0.5), ('emb_dropout_keep_prob', 0.3), ('embedding_initializer', <function embedding_initializer at 0x7ff86d796d08>), ('embedding_size', 300), ('hidden_size', 300), ('l2_reg_lambda', 1e-05), ('learning_rate', 1.0), ('max_steps', 40000), ('model_dir', 'trained_models/EARC-model-01'), ('num_classes', 19), ('num_epochs', 100), ('num_heads', 4), ('pos_embedding_size', 50), ('pos_vocab_size', 162), ('rnn_dropout_keep_prob', 0.3)] \n",
      "\n",
      "Model Directory: trained_models/EARC-model-01\n",
      "Dataset Size: 8000\n",
      "Batch Size: 20\n",
      "Steps per Epoch: 400.0\n",
      "Total Steps: 40000\n",
      "That is 1 evaluation step after each 600 training seconds\n"
     ]
    }
   ],
   "source": [
    "TRAIN_SIZE = train.shape[0]\n",
    "NUM_EPOCHS = 100\n",
    "BATCH_SIZE = 20\n",
    "EVAL_AFTER_SEC = 600\n",
    "TOTAL_STEPS = int((TRAIN_SIZE/BATCH_SIZE)*NUM_EPOCHS)\n",
    "\n",
    "model_dir = 'trained_models/{}'.format(MODEL_NAME)\n",
    "\n",
    "hparams  = tf.contrib.training.HParams(\n",
    "    embedding_initializer = embedding_initializer,\n",
    "    num_epochs = NUM_EPOCHS,\n",
    "    batch_size = BATCH_SIZE,\n",
    "    num_classes = len(TARGET_LABELS),\n",
    "    embedding_size = 300,\n",
    "    pos_embedding_size = 50,\n",
    "    pos_vocab_size = len(pos_idx),\n",
    "    emb_dropout_keep_prob = 0.3,\n",
    "    hidden_size = 300,\n",
    "    rnn_dropout_keep_prob = 0.3,\n",
    "    dropout_keep_prob = 0.5,\n",
    "    num_heads = 4,\n",
    "    attention_size = 50,\n",
    "    learning_rate = 1.0,\n",
    "    decay_rate = 0.9,\n",
    "    max_steps = TOTAL_STEPS,\n",
    "    l2_reg_lambda = 1e-5,\n",
    "    model_dir = model_dir\n",
    ")\n",
    "\n",
    "session_conf = tf.ConfigProto(allow_soft_placement=True, log_device_placement=True)\n",
    "session_conf.gpu_options.allow_growth = True\n",
    "\n",
    "run_config = tf.estimator.RunConfig(\n",
    "    session_config=session_conf,\n",
    "    log_step_count_steps=300,\n",
    "    save_checkpoints_steps=5000,\n",
    "    tf_random_seed=17081992,\n",
    "    model_dir=model_dir\n",
    ")\n",
    "\n",
    "print(hparams, \"\\n\")\n",
    "print(\"Model Directory:\", run_config.model_dir)\n",
    "print(\"Dataset Size:\", TRAIN_SIZE)\n",
    "print(\"Batch Size:\", BATCH_SIZE)\n",
    "print(\"Steps per Epoch:\",TRAIN_SIZE/BATCH_SIZE)\n",
    "print(\"Total Steps:\", TOTAL_STEPS)\n",
    "print(\"That is 1 evaluation step after each\", EVAL_AFTER_SEC, \"training seconds\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Estimator Training/Validation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "def serving_input_fn():\n",
    "    \"\"\"\n",
    "    Serving input function, should contain a receiver_tensor with the same features as the input during training, \n",
    "    this function will also be used during validation.\n",
    "    \"\"\"\n",
    "    receiver_tensor = {\n",
    "      'text': tf.placeholder(tf.string, [None]),\n",
    "      'e1': tf.placeholder(tf.string, [None]),\n",
    "      'e2': tf.placeholder(tf.string, [None]),\n",
    "      'p1': tf.placeholder(tf.string, [None]),\n",
    "      'p2': tf.placeholder(tf.string, [None]),\n",
    "    }\n",
    "    features = {\n",
    "      key: tensor\n",
    "      for key, tensor in receiver_tensor.items()\n",
    "    }\n",
    "    return tf.estimator.export.ServingInputReceiver(\n",
    "        features, receiver_tensor)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_spec = tf.estimator.TrainSpec(\n",
    "    input_fn = lambda: input_fn(\n",
    "        TRAIN_DATA_FILES_PATTERN,\n",
    "        mode = tf.estimator.ModeKeys.TRAIN,\n",
    "        num_epochs=hparams.num_epochs,\n",
    "        batch_size=hparams.batch_size\n",
    "    ),\n",
    "    max_steps=hparams.max_steps,\n",
    "    hooks=None\n",
    ")\n",
    "\n",
    "eval_spec = tf.estimator.EvalSpec(\n",
    "    input_fn = lambda: input_fn(\n",
    "        VALID_DATA_FILES_PATTERN,\n",
    "        mode=tf.estimator.ModeKeys.EVAL,\n",
    "        batch_size=hparams.batch_size\n",
    "    ),\n",
    "    exporters=[tf.estimator.LatestExporter(\n",
    "        name=\"predict\", # the name of the folder in which the model will be exported to under export\n",
    "        serving_input_receiver_fn=serving_input_fn,\n",
    "        exports_to_keep=1,\n",
    "        as_text=True)],\n",
    "    steps=None,\n",
    "    throttle_secs = EVAL_AFTER_SEC\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Removing previous artifacts...\n",
      "Experiment started at 16:32:40\n",
      ".......................................\n",
      "INFO:tensorflow:Using config: {'_model_dir': 'trained_models/EARC-model-01', '_tf_random_seed': 17081992, '_save_summary_steps': 100, '_save_checkpoints_steps': 5000, '_save_checkpoints_secs': None, '_session_config': gpu_options {\n",
      "  allow_growth: true\n",
      "}\n",
      "allow_soft_placement: true\n",
      "log_device_placement: true\n",
      ", '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 300, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7ff86d6732e8>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n",
      "Estimator Type: <class 'tensorflow_estimator.python.estimator.estimator.Estimator'>\n",
      "INFO:tensorflow:Not using Distribute Coordinator.\n",
      "INFO:tensorflow:Running training and evaluation locally (non-distributed).\n",
      "INFO:tensorflow:Start train and evaluate loop. The evaluate will happen after every checkpoint. Checkpoint frequency is determined based on RunConfig arguments: save_checkpoints_steps 5000 or save_checkpoints_secs None.\n",
      "\n",
      " * data input_fn:\n",
      "===========================================\n",
      "Input file(s): data/processed/train-*.tsv\n",
      "Batch size: 20\n",
      "Epoch Count: 100\n",
      "Mode: train\n",
      "Thread Count: 40\n",
      "Shuffle: True\n",
      "=========================================== \n",
      "\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "WARNING:tensorflow:From <ipython-input-27-2407fb8f52d4>:73: bidirectional_dynamic_rnn (from tensorflow.python.ops.rnn) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use `keras.layers.Bidirectional(keras.layers.RNN(cell))`, which is equivalent to this API\n",
      "WARNING:tensorflow:From /home/luis.magana/anaconda3/lib/python3.7/site-packages/tensorflow/python/ops/rnn.py:443: dynamic_rnn (from tensorflow.python.ops.rnn) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use `keras.layers.RNN(cell)`, which is equivalent to this API\n",
      "WARNING:tensorflow:From /home/luis.magana/anaconda3/lib/python3.7/site-packages/tensorflow/python/ops/rnn.py:626: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.cast instead.\n",
      "WARNING:tensorflow:From /home/luis.magana/anaconda3/lib/python3.7/site-packages/tensorflow/python/ops/metrics_impl.py:455: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.cast instead.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Saving checkpoints for 0 into trained_models/EARC-model-01/model.ckpt.\n",
      "INFO:tensorflow:loss = 7.8886385, step = 0\n"
     ]
    }
   ],
   "source": [
    "#if not RESUME_TRAINING:\n",
    "print(\"Removing previous artifacts...\")\n",
    "shutil.rmtree(model_dir, ignore_errors=True)\n",
    "#else:\n",
    "#    print(\"Resuming training...\")\n",
    "    \n",
    "tf.logging.set_verbosity(tf.logging.INFO)\n",
    "\n",
    "time_start = datetime.utcnow() \n",
    "print(\"Experiment started at {}\".format(time_start.strftime(\"%H:%M:%S\")))\n",
    "print(\".......................................\") \n",
    "\n",
    "import logging\n",
    "\n",
    "# get TF logger\n",
    "log = logging.getLogger('tensorflow')\n",
    "log.setLevel(logging.DEBUG)\n",
    "\n",
    "# create formatter and add it to the handlers\n",
    "formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')\n",
    "\n",
    "# create file handler which logs even debug messages\n",
    "fh = logging.FileHandler('tensorflow.log')\n",
    "fh.setLevel(logging.DEBUG)\n",
    "fh.setFormatter(formatter)\n",
    "log.addHandler(fh)\n",
    "\n",
    "estimator = create_estimator(run_config, hparams)\n",
    "\n",
    "tf.estimator.train_and_evaluate(\n",
    "        estimator=estimator,\n",
    "        train_spec=train_spec, \n",
    "        eval_spec=eval_spec\n",
    ")\n",
    "\n",
    "time_end = datetime.utcnow() \n",
    "print(\".......................................\")\n",
    "print(\"Experiment finished at {}\".format(time_end.strftime(\"%H:%M:%S\")))\n",
    "print(\"\")\n",
    "time_elapsed = time_end - time_start\n",
    "print(\"Experiment elapsed time: {} seconds\".format(time_elapsed.total_seconds()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Testing and Inference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "TEST_SIZE = 1393\n",
    "test_input_fn = lambda: input_fn(files_name_pattern= VALID_DATA_FILES_PATTERN, \n",
    "                                      mode= tf.estimator.ModeKeys.EVAL,\n",
    "                                      batch_size= TEST_SIZE)\n",
    "test_results = estimator.evaluate(input_fn=test_input_fn, steps=1)\n",
    "print(\"# Test Measures: {}\".format(test_results))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "myExamples = [\n",
    "    \"This is the sprawling <e1>complex</e1> that is Peru's largest <e2>producer</e2> of silver.\", #Other\n",
    "    \"The <e1>dog</e1> knocked over the monitor, and now is <e2>broken</e2>\", #what is broken? the dog? \n",
    "    \"<e1>Avocados</e1> come from farms in the <e2>south of Mexico</e2>\" #Entity-Origin\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "s = pd.Series(myExamples).apply(lambda x: text2features(x, MAX_DOCUMENT_LENGTH))\n",
    "feed = pd.DataFrame(s.tolist()).astype(str).to_dict(orient='list')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "export_dir = model_dir + \"/export/predict/\"\n",
    "\n",
    "saved_model_dir = export_dir + \"/\" + os.listdir(path=export_dir)[-1] \n",
    "\n",
    "print(saved_model_dir, \"\\n\")\n",
    "\n",
    "predictor_fn = tf.contrib.predictor.from_saved_model(\n",
    "    export_dir = saved_model_dir,\n",
    "    signature_def_key=\"prediction\"\n",
    ")\n",
    "\n",
    "output = predictor_fn(feed)\n",
    "print(output)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
